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DETAILED ACTION 

1. This action is responsive to communications: RCE and Preliminary amendments, 
both filed September 26, 2006 to the original application filed 08/09/01. 

2. Claims 20 - 39 are pending in this Office Action. 

Continued Examination Under 37 CFR 1.114 

3. A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 
09/26/2006 has been entered. 

Priority 

4. If applicant desires to claim the benefit of a prior-filed application under 35 U.S.C. 
1 19(e), a specific reference to the prior-filed application in compliance with 37 CFR 

1 .78(a) must be included in the first sentence(s) of the specification following the title or 
in an application data sheet. For benefit claims under 35 U.S.C. 120, 121 or 365(c), the 
reference must include the relationship (i.e., continuation, divisional, or continuation-in- 
part) of the applications. 

If the instant application is a utility or plant application filed under 35 U.S.C. 
1 1 1(a) on or after November 29, 2000, the specific reference must be submitted during 
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the pendency of the application and within the later of four months from the actual filing 
date of the application or sixteen months from the filing date of the prior application. If 
the application is a utility or plant application which entered the national stage from an 
international application filed on or after November 29, 2000, after compliance with 35 
U.S.C. 371, the specific reference must be submitted during the pendency of the 
application and within the later of four months from the date on which the national stage 
commenced under 35 U.S.C. 371(b) or (f) or sixteen months from the filing date of the 
prior application. See 37 CFR 1. 78(a)(2)(H) and (a)(5)(ii). This time period is not 
extendable and a failure to submit the reference required by 35 U.S.C. 119(e) and/or 
120, where applicable, within this time period is considered a waiver of any benefit of 
such prior application(s) under 35 U.S.C. 1 19(e), 120, 121 and 365(c). A benefit claim 
filed after the required time period may be accepted if it is accompanied by a grantable 
petition to accept an unintentionally delayed benefit claim under 35 U.S.C. 1 19(e), 120, 
121 and 365(c). The petition must be accompanied by (1) the reference required by 35 
U.S.C. 120 or 119(e) and 37 CFR 1.78(a)(2) or (a)(5) to the prior application (unless 
previously submitted), (2) a surcharge under 37 CFR 1.1 7(t), and (3) a statement that 
the entire delay between the date the claim was due under 37 CFR 1 .78(a)(2) or (a)(5) 
and the date the claim was filed was unintentional. The Director may require additional 
information where there is a question whether the delay was unintentional. The petition 
should be addressed to: Mail Stop Petition, Commissioner for Patents, P.O. Box 1450, 
Alexandria, Virginia 22313-1450. 
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If the reference to the prior application was previously submitted within the time 
period set forth in 37 CFR 1.78(a), but not in the first sentence(s) of the specification or 
an application data sheet (ADS) as required by 37 CFR 1.78(a) (e.g., if the reference 
was submitted in an oath or declaration or the application transmittal letter), and the 
information concerning the benefit claim was recognized by the Office as shown by its 
inclusion on the first filing receipt, the petition under 37 CFR 1.78(a) and the surcharge 
under 37 CFR 1.1 7(t) are not required. Applicant is still required to submit the reference 
in compliance with 37 CFR 1 .78(a) by filing an amendment to the first sentence(s) of the 
specification or an ADS. See MPEP § 201 .1 1 . Therefore, the amendment to the 
specification submitted May 8, 2006 to claim benefit of U.S. Provisional Application 
Serial No. 60/246,052 is not proper and will not be entered. 



Application/Control Number: 09/925,397 
Art Unit: 2162 



Page 5 



Declaration Under 35 USC § 131 

5. The evidence submitted September 12, 2006 is insufficient to establish diligence 
from a date prior to the date of reduction to practice of the "MediaNet: A multimedia 
Information Network for Knowledge Representation", In Conference on Internet 
Multimedia Management Systems. Vol. 4210 (numbered Pages 1-12), Boston, MA, 
Nov. 2000, 1 ST/SPIE.00, Author: Ana B. Benitez et al" reference to either a 
constructive reduction to practice or an actual reduction to practice. 

6. The following is a quotation of 35 U.S.C. 715.07(x) which forms the basis for 
diligence set forth in this Office action: 

What is meant by diligence is brought out in Christie v. Seybold, 1893 CD. 515, 64 O.G. 1650 (6th 
Cir. 1893). In patent law, an inventor is either diligent at a given time or he is not diligent; there are 
no degrees of diligence. An applicant may be diligent within the meaning of the patent law when he 
or she is doing nothing, if his or her lack of activity is excused. Note, however, that the record must 
set forth an explanation or excuse for the inactivity; the USPTO or courts will not speculate on 
possible explanations for delay or inactivity. See In re N'elson, 420 F.2d 1079, 164 USPQ 458 
(CCPA 1970). Diligence must be judged on the basis of the particular facts in each case. See MPEP 
§ 2138.06 for a detailed discussion of the diligence requirement for proving prior invention. 

7. A general allegation that the invention was completed prior to the date of the 
reference is not sufficient. Ex parte Saunders; 1883 CD. 23, 23 O.G. 1224 (Comm'r 
Pat. 1883). Similarly, a declaration by the inventor to the effect that his or her invention 
was conceived or reduced to practice prior to the reference data, without a statement of 
facts demonstrating the correctness of the conclusion, is insufficient to satisfy 37 CFR 
1.131. 

While conception is the mental part of the inventive act, it must be capable of proof, 
such as by demonstrative evidence or by a complete disclosure to another. Conception 
is more than a vague idea of how to solve a problem. The requisite means themselves 
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and their interaction must also be comprehended. See Mergenthaler v. Scudder, 1897 
CD. 724, 81 O.G. 1417 (D.C. Cir. 1897). 

8. At page 2, paragraph 2 of the affidavit, the claimant states: 

"Prior to November 6, 2000, we have conceived the claimed invention of the 
application. A Provisional Patent Application Serial No. 60/246,052 was filed on, 
November 6, 2000, with the USPTO. John R. Smith is the common inventor to the 
present application and the provisional '052 application. The '052 application as filed is 
attached herewith Exhibit A. The '052 application evidences our conception of the 
claimed invention prior to November 6, 2002, the date of the MediaNet reference." 

9. The evidence submitted is insufficient to establish applicant's alleged actual 
reduction to practice of the invention in this country or a NAFTA or WTO member 
country after the effective date of the Benitez reference. 

The following is a quotation of 35 U.S.C. 715.07 which forms the basis for the 
explanation of the affidavit set forth in this Office action: 

The affidavit or declaration and exhibits must clearly explain which facts or data applicant is relying on to 
show completion of his or her invention prior to the particular date. Vague and general statements in broad terms • 
about what the exhibits describe along with a general assertion that the exhibits describe a reduction to practice 
"amounts essentially to mere pleading, unsupported by proof or a showing of facts" and, thus, does not satisfy the 
requirements of 37 CFR 1.131(b). In re BorUowski, 505 F.2d 713, 184 USPQ 29 (CCPA 1974). Applicant must give a 
clear explanation of the exhibits pointing out exactly what facts are established and relied on by applicant. 505 F.2d at 
718-19, 184 USPQ at 33. See also In re Harry, 333 F.2d 920, 142 USPC 164 (CCPA 1964) (Affidavit "asserts that 
facts exist but does not tell what they are or when they occurred."). 

1 0. There was no indication as to which portions of Exhibit A of the affidavit are 
related to the various phrases in independent claims 20, 27 and 34 of the patent 
application. The elements of claims 20, 27 and 34 are not in Exhibit A. The applicant 
should provide a correlation on how various statements in the affidavit apply to these 
claims. 
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1 1 . The claim of benefit of a provisional patent application serial No. 60/246,052 
under 35 U.S.C. 119(e), is not in compliance with 37 CFR 1.78(a). 

For these reasons, the "Affidavit Under 37 C. F. R. § 1.131" has been disallowed. 

Claim Rejections - 35 USC § 103 

12. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 1 02 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 
were made absent any evidence to the contrary. Applicant is advised of the obligation 
under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 

13. Claims 20, 27 and 34 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Non-Patent Document: "Supporting Ranked Boolean Similarity Queries in MARS", 
IEEE Trans, on Knowledge and date Engineering, 10, Nov - Dec. 1998, Author: Ortega 
et al (hereinafter "Ortega") in view of Non-Patent Document: "MediaNet: A multimedia 
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Information Network for Knowledge Representation", In Conference on Internet 
Multimedia Management Systems. Vol. 4210 (numbered Pages 1 - 12), Boston, MA, 
Nov. 2000, 1 ST/SPIE.00, Author: Benitez et al (hereinafter "Benitez"). 

Regarding claims 20, 27 and 34, Ortega teaches a method for processing 
multimedia data in a computer system, comprising: 

receiving as input a high-level concept describing data to be accessed (see section 
1.2, page 4, paragraph 1, "A Boolean retrieval model (adapted for retrieval over images) 

is used to interpret the query being able to support such conceptual queries is 

critical for the versatility of large image databases.", and section 2, "In this section, we 
briefly describe the image features used . . . Other image features are available , 
however we restrict ourselves to queries involving only to the above features in this 
paper."); 

translating the high-level concept into a low-level query by using stored concept 
constructs which are defined using features derived from a plurality of application 
domains (see section 1.2, page 4, paragraph 1, "A Boolean retrieval model (adapted for 

retrieval over images) is used to interpret the query being able to support such 

conceptual queries is critical for the versatility of large image databases.", and section 2, 
"In this section, we briefly describe the image features used . . . Other image features 
are available , however we restrict ourselves to queries involving only to the above 
features in this paper."). 
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Ortega does not explicitly teach transferring the low-level query to one or more 
search engines and concept repository. 

Benitez teaches transferring the low-level query to one or more search engines to 
access information using the low-level query (see section 3, page 6, "Typical content- 
based retrieval systems indexed ... or the value for any low-level features in the image 

database" and section 4.2, page 9, "At this point and the results integrated into a 

unique list as described for visual queries"); and 

a concept repository for storing and accessing the concept constructs (see Fig.4 and 
section 4.1, page 7, "For each application, the list of concepts and relationships in the 
MediaNet knowledge based should be representative of the content in the database and 
the goal of the application task."). 

It would have been obvious to one of ordinary skill in the data processing art at the 
time of the present invention to combine teaching of the cited references because 
Benitez's teaching of "transferring the low-level query to one or more search engines to 
access information using the low-level query" and "concept repository" would have 
allowed Ortega's system to integrate both conceptual and perceptual representations of 
knowledge to impact a broad range of applications that deal with multimedia content at 
semantic and perceptual levels. This improves the performance of multimedia retrieval 
applications by using query expansion, refinement and translation across multiple 
content modalities as suggested by Benitez (see Abstract). 
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14. Claims 21 - 26, 28 - 33, and 35 - 39 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Ortega in view of Benitez and further in view of Non-Patent 
Document: "CAMEL: Concept Annotated image Libraries", In Storage and Retrieval for 
Image and Video Database, San Jose, CA, Jan. 2001. SPIE (numbered pages 1 - 12), 
Author: Natsev et al (hereinafter "Natsev"). 

Regarding claims 21, 28 and 35, Ortega teaches storing matching algorithms in a 
matching algorithm library module (see section 4.8, page 20 - section 4.8.3, page 25) 
and Benitez teaches storing the concept constructs in a concept library module (see 
Fig.4 and section 4.1, page 7, "For each application, the list of concepts and 
relationships in the MediaNet knowledge based should be representative of the content 
in the database and the goal of the application task."). 

Ortega or Benitez does not explicitly teach storing the features in a feature library 
module; and storing constraints in a constraint library module. 

However, Natsev further teaches storing the concept constructs in a concept library 
module (see section 1.2, "The concept cataloguing, or learning, phase is used to define 
visual concepts and build a concept library. Concepts are defined .... The concept 
library is module for a persistent storage of concepts"); 

storing the features in a feature library module (see fig. 3); and 

storing constraints in a constraint library module (see section 6, "Another 
improvement that we are considering is the introduction of spatial constraints in the 
query engine. For example, .... the corresponding document is ranked higher"). 
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It would have been obvious to one of ordinary skill in the data processing art at the 
time of the present invention to combine teaching of the cited references because 
Natsev's teaching of storing the concept constructs in a concept library module, storing 
the features in a feature library module and storing constraints in a constraint library 
module would have presented a new direction for improving current image query 
system, namely focusing the query specification part as gateway to better performance 
and usability (see Natsev section 6) for the combination Ortega and Benitez's system. 

Regarding claims 22, 29 and 36, Natsev teaches interfacing the library modules 
to the application domains (see fig. 3 and section 4, "Fig. 3 illustrates the architecture of 
the .... as well as the Concept Library module"). 

Regarding claims 23, 30 and 37, Natsev teaches building a concept construct 
(see fig. 1 and section 1.2, The concept cataloguing, or learning, phase is used to define 
visual concepts and build concept library. Concepts are defined . . . and to associate it 
with the given concept"). 

« 

Regarding claims 24 and 31, Natsev teaches a method as defined in Claim 23, 
wherein the step of building a concept construct comprise combining one or more of the 
features with see section 1.2, The concept cataloguing, or learning, phase is used to 
define visual concepts and build concept library) 
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zero or more of the stored concept (see Fig.1 and section 1.2, "The concept 
library is a module for persistent storage of concept"), and 

zero or more of the constraints (see section 6, "Another improvement that we are 
considering is the introduction of spatial constraint in query engine"). 

Regarding claims 25 and 32, Ortega teaches a concept construct is represented 
using a hierarchical fuzzy graph data tree-structure comprising: nodes that correspond 
to child-concepts and a subset of the features; aggregation edges that correspond to 
parent-child relationships; and association edges that correspond to inter-sibling 
constraints (see sections 4.1 and 4.2, pages 10 and 11). 

Regarding claims 26 and 33, Natsev teaches a method as defined in Claim 20, 
wherein the features are user defined (see section 2, "One of the most commonly used 
image features is color histogram, . . . places the burden on the user to specify weights 
for the different features"). 

Regarding claim 38, Natsev teaches a system as defined in Claim 34, wherein 
the translation engine further comprises an interpreter that translates the high level 
concept (see fig. 1 ). 

Regarding claim 39, Natsev teaches a system as defined in Claim 34, further 

comprises a search engine (see section 2, "the high-level approach that there are 

some considerations that make certain query engines more suitable than others". 
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Conclusion 



15. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Fred I. Ehichioya whose telephone number is 571-272- 
4034. The examiner can normally be reached on M - F 8:00 AM to 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John E. Breene can be reached on 571-272-4107. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



Fred I. Ehichioya 
Patent Examiner 
Art Unit 2162 




October 12, 2006 




